The present invention relates to an image processing technique for generating a synthesized image from images taken with a plurality of cameras. In particular, the present invention relates to a technique that can be effectively applied to monitoring systems used as an assist in ensuring safety driving of a vehicle and the like.
In recent years, with widespread use of car-mounted displays and cost reduction in video equipment such as cameras, devices for monitoring the surroundings of a vehicle with cameras to assist safety driving have been commercialized and are now becoming popular.
As an example of such conventional vehicle surroundings monitoring devices, the following construction is known. That is, a camera is placed at a position on a vehicle where an image of a blind area for the driver, such as the rear of the vehicle, can be taken with the camera, and the image taken (camera image) is shown on a display for monitoring.
For example, Japanese Laid-Open Patent Publication No. 58-110334 discloses the following construction. A plurality of cameras are placed at different positions on a vehicle. A plurality of camera images taken with the cameras are modified and synthesized, to generate a synthesized image as is viewed from the above of the vehicle with the vehicle in the center and display the synthesized image on an in-car display. Using this vehicle surroundings monitoring device, the driver sitting on the driver""s seat can monitor on the display a blind area of which direct view is not possible. This contributes to avoiding an accident that would otherwise have occurred, and also facilitating the driving.
[Problems to be Solved]
However, the inventors of the present invention have found from experiments and examinations that the conventional construction has the following problems.
FIGS. 22A to 22C illustrate examples of image synthesis performed when a vehicle is moving. In the case of using cameras for interlaced scanning, an input image as shown in FIG. 22A is obtained when the vehicle is moving. That is, every other horizontal line is displaced resulting in comb-shaped appearance. This input image is not particularly strange as long as it is displayed as it is on a display for interlaced scanning. However, when this input image is modified for synthesis, the comb-shaped displacement is also modified during the image modification as shown in FIG. 22B, and this causes a lag between the image-taking timing and the displaying timing. The resultant image makes the observer feel strange. That is, so-called interlace noise appears on the synthesized image more significantly.
The unnaturalness as described above does not appear when a one-field synthesized image is generated from one-field input images. However, the resolution of an input image is low in this per-field synthesis, compared with per-frame synthesis. Therefore, when an image of two fields is observed as a frame image as shown in FIG. 22C, comb-shaped displacement occurs due to an error in quantization of coordinate values even when the image is still. This deteriorates the image quality, especially in the area of a synthesized image where an input image has been enlarged, when the vehicle is standing still.
That is, as the first problem, when a one-frame synthesized image is generated from one-frame (two-field) input images, the synthesized image is unnatural when the vehicle is moving and when a moving image is taken. When a one-field synthesized image is generated from one-field input images, the resolution of the synthesized image is low.
As the second problem, in modification of a camera image, the input image may be contracted depending on the composition of a synthesized image. As a result, aliasing distortion may partially be generated in the synthesized image.
As the third problem, the junctures of input images on a synthesized image may be unnatural. For example, because white balancing and the like are adjusted separately for the respective cameras, the brightness and the tint may fail to match at the junctures as shown in FIG. 23A. Also, in some image synthesis methods, the junctures of images may be displaced due to calculation error, erroneous detection of the position of an object, and the like, as shown in FIG. 23B.
An object of the present invention is providing an image processor for generating a synthesized image from a plurality of camera images, which can generate a more natural synthesized image compared with the conventional processors.
Specifically, the image processor of the present invention, which receives camera images taken with a plurality of cameras capturing the surroundings of a vehicle and generates a synthesized image from the camera images, includes a parameter generation section constructed to be able to generate a plurality of image synthesis parameter groups each representing the correspondence between the camera images and the synthesized image and having different spatial or temporal resolution relations, wherein the synthesized image is generated from the camera images according to the image synthesis parameter group generated by the parameter generation section, and the parameter generation section switches the image synthesis parameter group to be generated according to an output of a vehicle motion detection section for detecting the motion of the vehicle.
According to the invention described above, it is possible to switch the image synthesis parameter group used for generation of a synthesized image according to the motion of the vehicle. This makes it possible to reduce the unnaturalness of the synthesized image observed when the vehicle is moving and when an image of a moving object is taken, and also prevent the reduction in resolution observed when the vehicle is standing still.
In the image processor of the present invention, the parameter generation section preferably includes: a parameter storage section for storing the plurality of image synthesis parameter groups; and a parameter selection section for selecting at least one among the plurality of image synthesis parameter groups stored in the parameter storage section according to the output of the vehicle motion detection section.
In the image processor of the present invention, preferably, the camera images are interlaced images, and the plurality of image synthesis parameter groups include at least a frame-base image synthesis parameter group and a field-base image synthesis parameter group. Preferably, the parameter generation section generates the field-base image synthesis parameter group when the motion of the vehicle detected by the vehicle motion detection section is relatively fast, and generates the frame-base image synthesis parameter group when the motion of the vehicle is relatively slow.
In the image processor of the present invention, the vehicle motion detection section preferably detects the motion of the vehicle from the camera images.
In the image processor of the present invention, preferably, the plurality of cameras are constructed to be able to switch a capture pattern according to an input switch signal, and the parameter generation section sends the switch signal to the cameras, together with generating the image synthesis parameter group, according to the output of the vehicle motion detection section, to switch the capture patterns of the cameras. Preferably, the parameter generation section selects the image synthesis parameter group and switches the capture patterns of the cameras according to an output of a vehicle status detection section for detecting brightness of the surroundings of the vehicle, in addition to the output of the vehicle motion detection section.
In the image processor of the present invention, preferably, the parameter generation section generates the image synthesis parameter group according to an output of a vehicle status detection section for detecting the vehicle status such as an operation by a driver of the vehicle or whether or not an obstacle exists in the surroundings of the vehicle, in addition to the output of the vehicle motion detection section.
The monitoring system of the present invention includes the image processor described above as an image processing section.
Alternatively, the image processor of the present invention, which receives camera images taken with a plurality of cameras capturing the surroundings of a vehicle and generates a synthesized image from the camera images, includes: a parameter storage section for storing a plurality of sets of an image synthesis parameter group representing the correspondence between the camera images and the synthesized image and a filter parameter group corresponding to the image synthesis parameter group; a parameter selection section for selecting at least one among the plurality of sets of the image synthesis parameter group and the filter parameter group stored by the parameter storage section according to an output of a vehicle motion detection section for detecting the motion of the vehicle and an output of a vehicle status detection section for detecting the status of the vehicle such as an operation by a driver of the vehicle or whether or not an obstacle exists in the surroundings of the vehicle; and a filtering section for performing frequency band limitation filtering for the camera images according to the filter parameter group of the set selected by the parameter selection section, wherein the synthesized image is generated from the camera images filtered by the filtering section according to the image synthesis parameter group of the set selected by the parameter selection section.
According to the invention described above, the filter parameter group is selected according to the motion and status of the vehicle, and the camera images are subjected to frequency band limitation filtering according to the selected filter parameter group. This effectively suppresses the aliasing distortion in the synthesized image.
In the image processor of the present invention described above, the filter parameter group preferably includes filtering setting data for each pixel position of the camera image.
The monitoring system of the present invention includes the image processor described above as an image processing section.
Alternatively, the image processor of the present invention, which receives camera images taken with a plurality of cameras capturing the surroundings of a vehicle and generates a synthesized image from the camera images, includes: a brightness correction parameter calculation section for calculating brightness correction parameters for correcting the brightness and tint of the camera images; and a brightness correction section for correcting the brightness and tint of the camera images using the brightness correction parameters calculated by the brightness correction parameter calculation section, wherein the synthesized image is generated from the plurality of camera images subjected to brightness correction by the brightness correction section according to an image synthesis parameter group representing the correspondence between the camera images and the synthesized image, the image synthesis parameter group includes overlap area data for an overlap area on the synthesized image in which coverages of the plurality of cameras overlap, the overlap area data indicating coordinates of pixels in camera images corresponding to the overlap area, and the brightness correction parameter calculation section receives the overlap area data and calculates the brightness correction parameters using brightness and tint data for the pixels in the camera images corresponding to the overlap area indicated by the overlap area data.
According to the invention described above, brightness correction parameters are calculated using information on the brightness and tint of the camera images corresponding to the overlap area on the synthesized image, and brightness correction is performed using the brightness correction parameters. This reduces the unnaturalness of the juncture on the synthesized image.
In the image processor of the present invention described above, preferably, the brightness correction parameter calculation section performs statistical processing on the brightness in the overlap area for the camera images corresponding to the overlap area, and calculates the brightness correction parameters based on the processing results.
In the image processor of the present invention described above, when a plurality of overlap areas exist, the brightness correction parameter calculation section preferably sets priorities to the overlap areas to be considered during the calculation of the brightness correction parameters according to an output of a vehicle motion detection section for detecting the motion of the vehicle.
In the image processor of the present invention described above, the brightness correction section is preferably incorporated in the cameras.
The monitoring system of the present invention includes the image processor described above as an image processing section.
Alternatively, the image processor of the present invention receives camera images taken with a plurality of cameras capturing the surroundings of a vehicle and generates a synthesized image from the camera images, wherein, in an overlap area in which coverages of a plurality of cameras overlap on the synthesized image, a camera image used for generation of the synthesized image is selected among camera images from the plurality of cameras according to an output of a vehicle motion detection section for detecting the motion of the vehicle or an output of a vehicle status detection section for detecting the status of the vehicle such as an operation by a driver of the vehicle or whether or not an obstacle exists in the surroundings of the vehicle.
Alternatively, the image processor of the present invention receives camera images taken with a plurality of cameras capturing the surroundings of a vehicle and generates a synthesized image from the camera images, wherein, in an overlap area in which coverages of a plurality of cameras overlap on the synthesized image, weights to camera images from the plurality of cameras are set according to an output of a vehicle motion detection section for detecting the motion of the vehicle or an output of a vehicle status detection section for detecting the status of the vehicle such as an operation by a driver of the vehicle or whether or not an obstacle exists in the surroundings of the vehicle.
In the image processor of the present invention described above, the processor includes: an image synthesis parameter group associating pixels in the synthesized image with pixels in the camera images; and a table representing the correspondence between a weight reference number and a combination of sets of weighting information, wherein a portion of the image synthesis parameter group corresponding to the overlap area holds any of the weight reference numbers shown in the table.
The monitoring system of the present invention includes the image processor described above as an image processing section.
Alternatively, the image processor of the present invention receives camera images taken with a plurality of cameras capturing the surroundings of a vehicle and generating a synthesized image from the camera images, wherein the image processor includes an image synthesis parameter group associating pixels in the synthesized image with pixels in the camera images, and in an overlap area in which coverages of the plurality of cameras overlap on the synthesized image, the image synthesis parameter group includes weights set to camera images from the plurality of cameras represented by a dithering method.